Time Variable Reinforcement Learning and Reinforcement Function Design

We introduce the mathematical model for time variable reinforcement learning. The policy, the rewards or reinforcement function and the transition probabilities may depend on the progress of the time t. We prove that under certain conditions slightly changed methods of classical dynamic programming assure finding the optimal policy. For that we deduct the Bellman equation for the time variable case and apply the fixed point theorem. Furthermore we present a particular flexible reinforcement function design frame with adjustable parameters and its theorical equivalence to general reinforcement functions. A parameter update algorithm (UPA) for the new reinforcement function in order to guarantee desired ratios of positive, negative and null rewards is introduced. In a series of real robot experiments we show that using the time variable reinforcement function introduced above may help to accelerate learning. Interesting results comparing the learning progress for wall following and an obstacle avoidance behavior implementing Q-learning and a radial basis function network are given. As a main result of our work we address the effectiveness of reinforcement function design and time variable reinforcement learning in general. CONTENTS About this work.......................................................................................................i Acknowledgement...................................................................................................i Abstract...................................................................................................................ii